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A FAST  SOLUTION  OF  THE  LINDLEY  EQUATIONS 
FOR  THE  M-GROUP  REGRESSION  PROBLEM* 


Ivo  W.  Molenaar 
University  of  Groningen 


0.  Summary 

This  report  is  based  on  a discussion  by  Jones  and  Novick  (1972)  of 
the  technical  problems  Involved  in  obtaining  Bayesian  modal  estimates 
for  the  regression  parameters  in  m similar  groups.  A general  discussion 
of  Bayesian  linear  h.^del  analysis  is  found  in  Lindley  and  Smith  (1972). 

The  methodology  of  m-group  regression  is  explained  in  Jackson,  Novick, 
and  Thayer  (1971).  Cross-validation  is  reported  by  Novick,  Jackson, 

Thayer  and  Cole  (1971,  1972),  and  Jansen  (1977);  Shigemasu  (1976)  pre- 
sents a solution  and  cross-validation  for  the  case  of  equal  slopes  and 
unequal  intercepts. 

The  available  computer  programs,  BPREP-BASIC  and  BAYREG,  both  written 
in  FORTRAN,  require  an  amount  of  core  and  CPU  that  does  not  encourage  regular 
use.  These  programs  have  been  analyzed  with  the  aim  of  suggesting  improvements 
that  might  bring  the  performance  of  algorithms  to  a point  where  inclusion  in  the 
CADA  Monitor  (Novick,  Isaacs,  and  DeKeyrel,  1977),  after  conversion  into  BASIC, 
could  become  feasible.  Slight  modifications  to  the  preparatory  program  BPREP, 

*This  research  was  supported  in  part  by  Office  of  Naval  Research  Con- 
tract #N00014-77-C-0428.  However,  any  opinions,  findings,  or  conclu- 
sions expressed  herein  are  those  of  the  author  and  do  not  necessarily 
reflect  the  views  of  the  funding  organization. 
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and  major  changea  to  BAYREG  were  made,  resulting  In  a speeded -up  algorithm. 
Compute  time  (both  on  the  IBM  370  VS  In  Iowa  City  and  on  the  CDC  Cyber  74/18 
In  Groningen)  was  reduced  by  a factor  of  2 to  4 in  four  benchmark  examples, 
which  are  believed  to  be  fairly  representative  of  the  type  of  multiple  re- 
gression problems  where  Bayesian  m-group  regression  might  be  used.  However, 
it  is  advisable  to  use  the  speeded  algorithm  for  some  more  data  sets  and 
build  up  more  experience  about  its  optional  parameters,  before  the  pro- 
gram is  included  in  the  CADA  Monitor. 

Section  1 of  this  report  can  be  read  as  an  update  to  the  input  des- 
cription in  Jones  and  Novlck  (1972),  giving  the  user  sufficient  informa- 
tion for  the  use  of  the  modified  programs.  Section  2 describes  the 
deficiencies  of  the  old  programs  BPREP  and  BAYREG,  and  briefly  outlines 
the  improvements  that  have  been  made  in  the  new  program  MBREG.  Section  3 
contains  the  detailed  proposal  for  a revision  of  BPREP  into  a new  prepara- 
tory program  MPREP  adapted  to  the  new  main  program  MBREG;  this  revision 
will  not  require  substantial  programing  efforts.  Section  4 is  devoted 
to  the  explanation  of  the  "leaps"  which  constitute  the  major  improvement 
of  MBREG  over  BAYREG.  The  performance  of  the  improved  program  for  five 
data  sets  is  described  in  section  5.  The  final  section  6 mentions  a few 
points  deserving  further  investigation;  some  of  these  could  be  settled 
before  the  CADA  Monitor  version  of  the  program  is  written. 
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1.  Revised  Input  Description 


Running  a Bayesian  m-Group  Analysis 


The  data  decks  for  MPREP  and  for  MBREG  are  quite  similar,  since  the 
data  deck  for  MPREP  Is  a subset  of  the  deck  for  MBREG.  Accordingly,  the 


data  deck  for  MBREG  is  described  below: 


1. 

Identification  Card 

(10A8) 

Col.  1-80 

Identification  for 

data 

2. 

Parameter  Card 

(314,  3F5.0,  712, 

3F5.1) 

M and  NV  must  be  read  in, 

other  parameters  get  default 

values  if  blank 

Col.  1-4  M Number  of  schools  (maximum  25) 

5-8  NV  Number  of  predictors  (maximum  4) 

9-12  NCY  Number  of  cycles  (maximum  100,  default  ■ 30) 

13-17  CKAPPA  Small  Constant  (default  = .001).  The  constant 

k in  the  equations  is  taken  as  CKAPPA/THETA, 
where  THETA  is  the  harmonic  mean  of  the 
(after  replacing  < PHIMIN  by  this  lower 
bound) 


18-22 

DCON 

X • DCON  t (default  = 3) . The  diagonal  of  Z 
in  the  equations  contains  DCON  * (a  priori 
estimates  of  t and  rg.  given  on  card  9) 
Minimum  $ allowed  (cln  be  arbitrarily  small, 
default  = .001) 

23-27 

PHIMIN 

28-29 

IWR 

0 - No  details  on  iteration  (default) 

1 - Criterion  and  all  values  printed  after 
each  cycle 

30-31 

INIT 

0 - Starting  values  in  original  scaling  of 

card  3 (default) 

1 - Starting  values  in  ideal  scaling 

2 - Starting  values  scaled  at  origin 

32-33 

IPUN 

0 - No  punched  output  (default) 

1 - Bayesian  modal  estimates  are  punched 

(8X,  6E12.6) 

34-35 

NDH 

Iteration  stops  when  criterion  constant  in 

NDH  digits  (default  " 5) 

36-37 

NDB 

Iteration  stops  when  all  a.,  8,  .,  and  $ con- 
stant in  NDB  digits  (default  *4);  if  rela- 
tive change  is  less  than  10”*®®  no  leap  is 
taken  for  that  estimate 

38-39 

NCI 

Number  of  cycles  before  first  leap  (mini- 
mum 4,  default  ■ 5) 

40-41 

NCF 

Number  of  cycles  between  leaps  (minimum  4, 
default  * 4) 

i 
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2.  Parameter  Card  (con't.) 
Col.  42-46  SCH 


47-51  DCN 


52-56  VGT 


3.  Predictor  Card 
Col.  1-8 
9-16 


Leap  - SCH  * last  difference  if  difference 
has  just  changed  sign  or  old  difference  al- 
most equal  to  0 (default  - 20.0) 

Leap  - DCN  * last  difference  if  this  differ- 
ence is  not  substantially  closer  to  zero 
than  previous  difference  (default  - 10.0) 

At  first  decrease  of  criterion  not  after  leap 
or  start,  next  cycle  starts  at  VGT  * old  + 

(1  - VGT)  * new  values  (default  * .5) 

(4A8) 

Name  of  1st  predictor 
” " 2nd  " 


4.  Scaling  Card  for  Original  Scaling  (5F8.0) 
Points 

Col.  1-8  Value  t 

9-16  Value  t 

17-24  " 


Value  to  which  criterion  has  been  scaled 
Value  to  which  predictor  1 has  been  scaled 

»i  n ii  ii  2 " ’I  ii 


5.  Scaling  Card  for  Ideal  Points 
Col.  1-13 
14-26 
27-39 


(5E13.6) 

Value  to  which  criterion  has  been  scaled 
Ideal  scaling  point  for  predictor  1 

H «!  11  11  II  2 


6.  Format  Card  for  SCP  Matrix  (A8) 

The  cross  products  must  be  read  in  floating  point  form. 

7.  SCP  Matrix  Cards 

For  each  group,  there  must  be  an  upper  triangular  cross-product  matrix 
punched  according  to  the  format  specified  by  card  6.  The  cross-product 


matrices  have  the  following  form  for  the  case  of  two  predictors: 


Row  1 


Row  2 


Row  3 


£xllJ 

£xi2J 

ExilJx12j 

£xilj5'lj 

£xi2J^iJ 

Row  4 


Comparison  of  this  matrix  with  formula  (5)  reveals  that  rows  2,  3,  and 
4 have  been  translated  to  the  left.  These  cross  products  are  scaled  to  the 
values  given  by  card  4. 

8.  Coefficient  cards  (6E12.6) 

For  each  group,  there  must  be  an  initial  coefficient  vector  which  con- 
sists of  (a^,  ...»  8^,  The  is  for  the  data  coded  to  origi- 

nal scaling,  ideal  scaling  or  scaling  at  origin  for  INIT  * 0,  1,  2 respec- 
tively. In  most  cases  the  coefficient  cards  will  contain  the  least  square 
coefficients  per  group,  produced  by  MPREP  or  obtained  separately. 

9.  Population  Variance  Estimates  Card  (6E12.6) 

See  Jones  and  Novick  (1972,  pp.  17-18).  The  estimates  for  population 

variances  are  the  components  of  (x  , t , ...»  ?o0).  Typically  t and  x0, 

oi  pi  oi  pn 

are  values  computed  by  the  MPREP  program;  if  some  of  the  original  estimates 
were  negative,  then  these  estimates  have  been  replaced  by  a small  positive 
number.  The  value  (at  ideal  point)  for  asymptotic  values  has  then  been 
recalculated. 


The  data  deck  for  MPREP  differs  from  that  for  MBREG  in  the  following 
ways.  The  scaling  card  for  ideal  points  is  not  included  nor  are  the  co- 
efficient cards  or  the  population  statistics  (all  these  cards  result 
from  the  MPREP  analysis).  In  summary,  the  respective  items  in  the  data 
decks  are  listed  below. 


MPREP 

MBREG 

1. 

Identification  Card 

Yes 

Yes 

2. 

Parameter  Card 

Yes 

Yes 

3. 

Predictor  Card 

Yes 

Yes 

4. 

Scaling  Card  for  Original  Scaling  Points 

Yes 

Yes 

5. 

Scaling  Card  for  Ideal  Points 

No 

Yes 

6. 

Format  Card  for  SCP  Matrix 

Yes 

Yes 

7. 

SCP  Matrix  Cards 

Yes 

Yes 

8. 

Coefficient  Cards 

No 

Yes 

9. 

Population  Variance  Estimates  Card 

No 

Yes 

Items  8 and  9 are  provided  as  part  of  the  punched  output  from  MPREP. 
Item  5 i3  the  first  card  of  this  output.  (In  the  earlier  version  it  was 
punched  by  the  investigator  from  the  printed  output  from  MPREP  and  inserted 
in  the  data  deck. 

Several  bits  of  information  required  in  items  1-9  can  be  placed  in 
the  context  of  Bayesian  methodology.  The  constant  CKAPPA  bounds  the  pos- 
terior density  away  from  the  point  at  which  the  residual  variances  are 


equal.  The  value  DCON  is  a factor  which  inflates  the  estimates  of  r ; 

P 

it  formally  equates  the  mode  of  the  prior  to  the  value  specified  a priori 
(cf.  Jackson,  Novick,  and  Thayer,  1971,  p.  131).  PHIMIN  places  a lower 
bound  on  the  residual  variance  estimate  for  any  given  group. 

Often  it  is  desirable  to  set  the  value  of  IWR  to  one,  since  the 
investigator  will  then  be  able  to  observe  the  computer's  search  for  the 
mode  of  the  posterior  distribution.  (The  reason  that  the  log  of  the 
height  function  is  printed  rather  than  the  actual  height  is  that  storage 
of  the  actual  height  might  cause  problems  within  the  computer  when  the 
height  is  very  small.)  As  the  iteration  process  converges,  the  log  of 
the  height  function  will  Increase  to  the  modal  value  and  then  exhibit 
very  small  fluctuations  about  this  value. 

The  scaling  values  for  the  original  scaling  points  (item  4)  may  be 
all  zero  (indicating  lack  of  rescaling)  if  desired;  however,  if  the  in- 
vestigator uses  variables  with  large  means,  he  may  wish  to  scale  the 
variables  to  their  approximate  means  before  producing  the  SCP  matrix. 

To  do  this,  he  simply  subtracts  a constant  from  each  individual's  score. 
Such  a procedure  will  tend  to  reduce  the  magnitude  of  the  numbers  con- 
tained in  the  SCP  matrix  and  may  decrease  the  chances  for  rounding  errors 
in  a problem  involving  large  sample  sizes.  A different  scaling  point,  of 
course,  may  be  selected  for  each  variable. 


The  original  scaling  points  should  be  distinguished  from  the  ideal 
scaling  points.  The  ideal  scaling  points  are  not  a mere  matter  of  con- 
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venience;  they  are  intrinsic  to  the  Bayesian  method  being  discussed. 
Also,  the  ideal  points  apply  only  to  the  predictor  variables,  whereas 
the  original  scaling  points  are  relevant  to  the  predictors  and  to  the 
criterion. 

We  punch  the  symmetric  SCP  matrix  in  a special  upper-triangular 
form.  The  first  row  contains  the  full  number  of  (£  + 2)  elements. 

If  necessary,  the  punching  continues  on  subsequent  cards  according  to 
the  format  card.  The  second  row  must  begin  on  a new  card  but  omits 
the  first  element  in  order  to  begin  with  the  diagonal  elements  in  the 
second  row;  thus,  the  punched  form  of  the  second  row  has  only  (i  + 1) 
elements.  Continuing  in  this  fashion,  we  find  that  the  last  rwo,  when 
punched,  contains  only  the  diagonal  (last)  element. 

We  next  present  some  recommendations  for  the  input  parameters  gov- 
erning the  leap  process  which  is  discussed  in  more  detail  in  section  4. 
When  a user  has  no  special  information  on  his  m-group  regression  data 
set,  it  is  recommended  that  he  start  running  the  program  with  all  de- 
fault options  in  vigor.  The  log  posterior  density  is  printed  after 
every  cycle,  and  gives  an  impression  of  the  convergence  of  the  estima- 
tion process.  When  desired,  the  option  IWR  * 1 leads  to  the  printing  of 
criterion  and  parameter  values  at  every  cycle  (one  line  per  group  per 
cycle) , which  permits  a detailed  investigation  of  the  behavior  of  the 
individual  parameters  during  iterations  and  leaps. 

If  these  results  are  not  satisfactory,  it  is  recommended  to  look  in 
some  detail  at  the  first  leap.  If  this  leap  is  based  for  a majority  of 
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parameter  values  on  differences  which  exhibit  a sign  change  or  an  in- 
crease in  absolute  value  (see  examples  in  section  4),  it  may  be  wiser 
to  postpone  the  first  leap  for  a few  cycles  by  choosing  a larger  value 
for  NCI.  When  it  appears  that  the  leaps  are  too  small  (l.e.,  the  itera- 
tions following  it  move  the  estimates  in  the  same  direction);  then  it  may 
help  to  increase  SCH  and/or  DCN,  provided  that  at  least  one  leap  followed 
after  a sign  change  or  increase  of  differences.  Similarly  a low  value 
for  DCN  and/or  SCH  could  be  inserted  if  the  leaps  are  too  large  (are 
followed  by  movements  in  the  opposite  direction) . It  should  be  stressed 
that  these  manipulations  have  a direct  effect  on  the  estimates  leaping 
after  sign  change  or  increase  of  differences,  but  also  an  indirect  effect, 
via  the  mean  across  groups,  on  all  other  estimates.  The  very  large  value 


DCN  = 30.0  was  successful  for  three  of  the  data  sets  mentioned  in  section  5 
for  just  this  reason:  it  produced  a too  large  leap  for  the  relatively  few 


parameters  where  the  difference  had  not  decreased,  but  in  the  next  itera- 
tion step  this  pulled  all  the  remaining  estimates  which  had  leapt  not  far 
enough  by  the  geometric  ratio  extrapolation,  in  the  right  direction  more 
than  an  ordinary  Iteration  would  have  done.  As  this  phenomenon  might  be 
unusual,  we  have  kept  the  safer  value  DCN  - 10  as  default  value. 

Part  of  the  success  of  large  DCN  values  is  explained  by  the  rule  in 
the  program  that  an  estimated  geometric  ratio  P (see  section  4)  for  which 
p/ (1  - p)  exceeds  DCN  is  replaced  by  DCN.  This  rule  prevents  gigantic 
overshoots  for  cases  where  a ratio  p * .98,  say,  is  produced  at  a point 


the  process  to  continue  when  a user  might  be  satisfied  with  the  precision 
of  the  estimates.  This  holds  especially  when  high  precision  is  desired. 
Moreover,  optimal  speeding  of  the  leap  process  is  sometimes  impeded  by 
a choice  NDB  * 2 or  NDB  * 3 which  might  work  well  in  the  stopping  rule. 
Perhaps  the  double  role  should  be  revised  when  more  experience  on  other 
data  sets  has  been  collected. 

The  undisturbed  iteration  process,  without  leaps  or  reduced  steps, 
is  obtained  when  the  user  specifies  values  NCT  >_  NCY,  NCI  >_  NCY  and 
VGT  - 0.0.  / detailed  study  of  this  process  (IWR  ■ 1)  may  be  desirable 

when  the  speeded  solution  process  does  not  produce  satisfactory  results. 
Such  cases  have  not  yet  been  encountered,  and  the  "bad  leap"  provision 
already  offers  protection  against  occasional  malfunctioning,  but  it  may 


t 
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still  be  worthwhile  to  know  that  the  original  slow  convergence  process, 
without  leaps  and  without  other  stopping  rules  than  NCY  * total  number  of 
cycles,  is  still  available  as  a subset  in  the  present  program. 

By  putting  IPUN  = 1,  the  user  obtains  the  best  solution  in  punched 
cards.  This  may  be  useful  when  a restart  from  this  solution  with  other 
leap-governing  parameters  is  desired;  it  also  saves  computer  time  in 
cases  where  the  process  might  turn  out  to  be  "on  its  way"  without  having 
achieved  yet  the  amount  of  convergence  required  by  the  user. 

2.  Conclusions,  Def i''.iencies  and  Changes  in  the  Programs 

Changes  were  made  in  BAYREG  and  proposed  for  BPREP  with  the  double 
aim  of  speeding  the  execution  and  reducing  the  core  requirements.  Study 
of  both  programs  in  their  July  1977  form  led  to  the  following  conclusions: 

1.  BAYREG  either  uses  two  sets  of  initial  estimates,  least  squares  (LS) 
and  asymptotic  (ASY,  essentially  Model  II  analysis  of  variance  esti- 
mates; see  Jones  and  Novick,  1972,  p.  18),  or  uses  read-in  initial 
estimates. 

2.  The  final  Bayesian  estimates  maximizing  the  criterion  (which  is  the 
logarithm  of  the  posterior  density)  are  typically  closer  to  ASY  than 
to  LS,  and  the  criterion  value  for  ASY  is  higher  than  for  LS.  Never- 
theless, iteration  from  LS  starting  values  produced  in  all  benchmark 
examples  a sequence  of  criterion  values  which  overtook  the  sequence 
starting  at  ASY  values,  sometimes  at  the  third  sometimes  at  the  fifth 
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cycle.  Criterion  values  from  LS  starting  remained  consistently  higher 
during  the  further  iteration  process.  Novick,  Jackson,  Thayer,  and 
Cole  (1971,  p.  27)  have  also  started  iteration  from  LS  values. 

3.  In  the  iterative  solving  of  the  Lindley  equations,  BAYREG  uses  as 
prior  values  three  times  the  estimated  population  variances  (of 
the  intercept)  and  Tgh  (of  the  h^  predictor,  h - 1,  2,  . . . , l) . 

It  takes  those  as  calculated  and  punched  in  BPREP,  but  changes  nega- 
tive values  into  10  ^ and  then  adapts  x^  to  such  changes.  This 

leads  to  totally  regressed  values  for  all  h for  which  x„,  was 

hi  ph 

negative  and  to  changes  in  x^  and  values.  This  modified  ASY  solu- 

tion is  used  as  a starting  point  for  the  iteration. 

4.  BAYREG  contains  dimension  statements  for  some  arrays  not  used  at  all, 
and  frequently  uses  two  or  more  arrays  where  one  array  could  serve  for 
more  than  one  purpose.  Moreover,  the  chaotic  numbering  of  the  state- 
ments and  the  use  of  parallel  names  for  essentially  the  same  variable 
afflicts  the  transparency  of  the  program.  It  Is  clear  that  the  program 
could  use  a "cleaning"  after  frequent  adhoc  changes. 

5.  Either  immediately  or  after  a few  cycles,  the  iteration  process  typically 
consists  for  many  cycles  of  taking  too  small  steps  in  the  right  direction. 

6.  After  finally  reaching  a kind  of  plateau  value  for  the  criterion,  the 
iteration  process  shows  an  oscillatory  behavior,  in  which  a step  lead- 
ing to  a small  decrease  of  the  criterion  is  followed  by  some  steps 
bringing  it  back  to  roughly  the  best  value  obtained  thus  far.  Near  its 
maximum,  the  criterion  seems  to  be  a very  flat  function  of  its  many  variables. 
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details  on  the  necessary  revision  of  MPREP  are  given  in  section  3. 

This  preparatory  program  should  be  adapted  to  produce  all  the  punched 
output  that  a user  might  need  for  the  input  data  deck  to  MBREG.  In 
the  stage  of  insertion  into  the  CADA  Monitor,  the  preparatory  program 
and  the  MBREG  program  can  be  turned  into  two  separate  modules  with  a 
division  point  at  which  only  a limited  data  transfer  is  required  and 
with  a more  logical  division  of  tasks  between  the  two  modules.  Some 
remarks  on  the  use  and  specification  of  the  prior  variances  will  be 
made  in  section  6 of  this  report. 

C.  The  program  BAYREG  has  been  completely  rewritten,  in  a more  accessible 
notation  involving  less  arrays  and  variables.  From  14401  array  ele- 
ments in  BAYREG  there  are  left  2109  elements  in  the  rewritten  program 
MBREG;  the  number  of  FORTRAN  lines  was  reduced  from  429  to  336,  in- 
cluding many  more  comment  lines  than  before.  This  will  mean  faster 
compilation,  less  core  requirements,  and  faster  execution.  Part  of 
this  economy  is  achieved  by  reducing  the  maximum  number  of  predictors 
from  10  to  4 and  the  maximum  number  of  groups  from  40  to  25.  This 
harmonizes  with  the  pleas  of  Jones  and  Novlck  (1972,  pp.  10-11)  and 
Jackson,  Novick,  and  Thayer  (1971,  p.  139).  The  revised  program  is 
designed  to  make  further  reduction  or  extension  of  these  limits,  when 
desired,  as  easy  as  possible. 

. 
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D.  The  slow  climbing  mentioned  in  conclusion  (5)  Is  speeded  up  by  taking 
a "leap"  to  values  extrapolated  from  the  values  at  the  past  three 
cycles.  Such  leaps  are  taken  every  NCF  cycles  (typically  5 or  4), 
with  an  Initial  leap  after  NCI  cycles.  The  leap  process  Is  goveri  ed 
by  several  parameters  described  In  sections  1,  4,  and  5.  Default 
values  for  these  parameters  are  provided  by  the  program,  but  the  user 
may  provide  other  values  when  the  default  values  do  not  lead  to  a 
satisfactory  convergence  of  the  iteration  process.  The  graphs  given 
in  section  5 clearly  show  for  some  benchmark  examples  that  the  intro- 
duction of  such  leaps  reduces  the  required  number  of  iterations  by  a 
factor  of  at  least  3. 

E.  Conclusion  (6)  has  led  to  the  Introduction  of  a special  provision  for 
any  Iteration  which  would  lead  to  a decrease  of  the  criterion  to  be 
maximized,  which  is  log  posterior  density.  The  parameter  values  to 
be  maximized  are  stored  before  the  decrease  and  the  step  Is  reduced 
to  1 - VGT  times  its  size  (VGT  ■ .5  Is  usual).  If  Iterations  resumed 
from  this  point  in  the  parameter  space  do  not  Improve  upon  the  maximum 
criterion  value  thus  far  within  the  next  two  cycles,  the  process  stops 
and  the  maximizing  values  are  printed.  Otherwise,  the  iteration 
process  continues  but  no  leaps  are  undertaken  until  NCF  iterations 
have  been  made  from  the  values  reached  by  the  reduced  step. 
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F.  The  process  stops  after: 

- a failure  of  Improvement  occurs  as  described  under  (E) ; 

- all  parameter  values  undergo  a relative  change  of  less  than  10  1108 
(remain  constant  In  NOB  leading  digits); 

- the  criterion  undergoes  a relative  Increase  of  less  than  10  10)11 
(remains  constant  or  decreases  In  NDH  leading  digits)  during  several 
cycles,  preceded  by  a leap; 

- a maximum  of  NCY  cycles  has  been  carried  out  (default  value  NCY  = 30, 
maximum  NCY  is  100;  inspection  and  revision  of  the  iteration  process 

is  believed  to  be  more  fruitful  than  prolonged  ineffective  iterations) . 
In  all  cases,  the  best  modal  estimates  obtained  thus  far  are  printed 
(and  also  punched  vhen  desired). 

G.  The  (alphabetical)  ordering  "alpha,  beta,  phi"  is  systematically  used  in 
all  inputs  and  outputs. 

H.  It  will  be  recommended  in  section  6 to  make  the  CADA  Monitor  version  of 
the  program  more  fool-proof  with  respect  to  conclusion  (7) . 

3.  Changes  in  the  Preparatory  Program  BPREP 

Aa  documented  in  Jones  and  Novick  (1972),  the  preparatory  program  in 
its  present  form  serves  three  purposes: 

(i)  It  calculates  the  "ideal  scaling  points,"  i.e.,  the  values  at  which 
the  predictors  must  be  centered  in  order  to  make  them  a priori  uncor- 
related with  the  intercept.  These  ideal  scaling  points  and  the  es- 
timated predictor  variances  and  covariances  between  intercept  and 
predictors  are  printed. 
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(li)  It  calculates  the  least  squares  (LS)  estimates  of  regression  co- 
efficients and  intercepts,  together  with  their  estimated  squared 
errors,  and  the  estimated  residual  variances.  These  are  all  punched. 
The  printed  output  omits  the  estimated  squared  errors,  and  adds  the 
intercept  values  for  predictors  scaled  at  the  origin,  the  scaling 
point,  arid  the  ideal  point. 

(iii)  It  calculates  the  asymptotic  (ASY)  estimates,  essentially  Model  II 
analysis  of  variance  estimates,  of  the  regression  coefficients,  the 
intercepts,  and  the  residual  variances.  The  variances  of  the  re- 
gression coefficients,  calculated  across  groups,  involve  both  LS  and 
ASY  estimates  as  detailed  by  Jones  and  Novick  (1972,  pp.  17-18). 

The  estimates  fg  obtained  in  this  way  can  be  negative;  they  are 
printed  and  punched  together  with  x . 

In  the  July  1977  form  of  BAYREG,  from  statement  99  onward,  these  re- 
sults are  used  as  follows: 

a.  Possible  negative  estimates  Xg^  are  replaced  by  10  ^ . (The  lower  bound 
-4 

of  10  given  by  Jones  and  Novick,  1972,  p.  18,  can  be  too  high  to 

produce  the  almost  total  regression  of  the  coefficients  3.  ; in  our 

n 

opinion  the  lower  bound  should  be  made  dependent  on  the  order  of  mag- 
nitude of  the  coefficients  B,  themselves,  because  a general  purpose 

h 

program  should  be  robust  against  rescaling  of  predictors  or  criterion, 
see  also  section  6.) 
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b.  Possible  residual  variance  estimates  $ less  than  PHIMIN  are  replaced 
by  PHIMIN  (a  lower  bound  which  can  be  arbitrarily  small,  according  to 
Jones  and  Novick,  1972,  pp.  22  and  25).  We  shall  point  out  in  sec- 
tion 6 that  such  a bound  should  be  robust  against  rescaling. 


c.  The  ASY  estimates  o and  B are  calculated  as  in  the  equations  on 


top  of  page  18  of  Jones  and  Novick  (1972)  but  with  omission  of  the 


-U 


factors  (1  - m ) . 


The  estimated  intercepts  variance  is  recalculated  for  the  revised 


ASY  estimates  a^. 


The  ASY  estimates  ^ for  the  residual  variances  are  obtained  from  the 


LS  estimates  in  the  following  way: 


1/^  - 1 


,th 


(n^  = sample  size  of  i group) ; 


log  <p±  + p±; 


m 


t **  m-1  I t ; 
i=l 


m 


m 


C - E <t  - t)  / (m  - 1)  - 2 I p./m; 
i-1  1 i-1  1 


exp{(Ct1  + 2p±t)/(C  + 2p±)} 


if  C > 0; 
otherwise. 


(.  exp(t) 

One  recognizes  the  weighted  average  character  of  such  estimates,  and 
it  will  be  obvious  that  the  averaging  and  weighting  is  carried  out 
for  the  logarithm  of  the  variance  rather  than  for  the  variance  itself. 
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7.  The  user  is  asked  to  provide  a minimum  value  for  the  residual  vari- 
ance (PHIMIN) , and  for  the  prior  parameter  variance  (t  ) , and  he  may 

pn 

use  card  4 to  choose  wise  ze-o  values  for  the  predictand  and  the  pre- 
dictors. Malfunctioning  may  occur,  however,  when  such  choices  would 


I 


inadvertently  be  made  in  total  disagreement  with  the  characteristics 
of  the  data. 


On  the  basis  of  these  conclusions,  several  program  changes  have  been 
carried  out.  They  are  summarized  below;  where  necessary  a more  detailed 
description  and  discussion  is  presented  in  the  next  sections. 

A.  Because  of  conclusions  (1)  and  (2),  the  new  MBREG  program  uses  just 
one  set  of  Initial  estimates.  In  most  cases  these  will  be  the  least 
squares  (LS)  solutions  for  the  individual  groups.  The  user  may,  how- 
ever, introduce  the  ASY  estimates  obtained  from  MPREP,  the  final  es- 
timates obtained  at  an  earlier  run  of  MBREG,  or  any  other  set  he  de- 
sires to  use.  In  the  first  two  cases  these  estimates  are  produced  in 
punched  cards  when  desired,  ready  for  introduction  into  MBREG.  In 
the  third  case  manipulation  of  the  INIT  parameter  (see  section  1)  al- 
lows the  use  of  intercept  values  corresponding  to  any  scaling  of  the 
predictors. 

B.  The  adaptation  of  the  prior  variances  t and  t„.  (h  ■ 1,  2,  . . .,  i) 

a eh 

mentioned  in  conclusion  (3)  will  be  made  in  the  MPREP  program.  This 
means  ASY  estimates  will  no  longer  be  calculated  in  the  MBREG  program. 


which  then 


requires  less  input  and  less  preparatory  calculations.  More 


MiMI 
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For  a convenient  use  of  the  new  MBREG  program,  the  new  MPREP  should 
produce  punched  output  which  can  Immediately  serve  as  Input  to  MBREG. 

From  the  Input  description  In  section  2,  It  can  be  seen  that  this  means 
punching  of  the  cards  1,  2,  . . .,5  (of  which  the  user  might  want  to 
adapt  the  parameter  card  to  his  wishes  if  he  is  not  satisfied  with  the 
default  values,  and  of  which  the  ideal  points  scaling  card  is  the  only  one 
containing  new  information) . The  input  to  MPREP  would  have  reversed  order 
of  predictors  and  criterion  on  card  4,  compared  to  BPREP,  in  order  to  be 
compatible  with  MBREG.  The  deck  of  SCP  matrix  cards  can  be  taken  from  the 
MPREP  input,  including  its  format  card  for  which  10A8  has  been  replaced 
by  A8.  The  coefficient  cards  mentioned  under  8 in  section  1 should  be 
punched  by  MPREP  in  the  new  format,  omitting  the  estimated  squared  errors 
cards. 

If  ASY  estimates  are  considered  to  be  important,  all  steps  (a)  to  (e) 
will  have  to  be  incorporated  in  the  MPREP  program.  If  not,  there  remains 
the  problem  that  input  card  9 of  MBREG  requires  population  variances  es- 
timates t and  r (h  = 1,  2,  . . .,  A).  They  could  be  printed  and  punched 
ct  pn 

as  in  the  present  BPREP  program  without  substantial  changes  if  the  revision 
of  t according  to  (d)  is  neglected.  We  shall  discuss  in  section  6 the 
whole  problem  of  the  r estimates;  pending  further  investigation  it  will  be 
left  open  whether  it  is  worthwhile  to  incorporate  a revised  estimate  of 
t from  the  data  into  MPREP. 


ft 
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4.  How  Leaps  Speed  Up  Convergence 

This  section  tells  how  end  why  leaps  have  been  Included  in  the  main 
program  MBREG.  An  illustration  of  the  argument  is  borrowed  from  the  next 
section,  in  which  the  method  will  be  applied  to  several  data  sets. 

Table  1 (see  appendix)  describes  in  some  detail  the  estimated  values  of  the 
intercept  (with  predictors  centered  at  the  ideal  point),  the  regression 
coefficients  6^  (h  - 1,  2,  3,  4)  and  the  residual  variance  for  the 
first  of  10  groups  for  which  it  was  tried  to  predict  grade  point  average 
from  the  four  predictors  English,  Mathematics,  Social  Sciences,  and 
Natural  Sciences.  The  data  set  is  described  under  the  name  "10HETERO" 
in  section  5.  During  the  iteration  process,  the  parameter  values  change 
more  or  less  systematically,  after  the  first  few  steps;  the  Improvements 
become  smaller  and  smaller,  and  the  parameters  seem  to  undergo  a monotone 
and  decelerating  movement  toward  the  optimal  values.  A similar  behavior 
is  observed  with  any  other  group  in  the  "10HETERO"  data  set,  and  with  the 
groups  of  the  other  data  sets  that  have  been  analyzed.  This  phenomenon 
is  familiar  to  many  numerical  analysts,  and  has  been  described  by  Fischer 
(1974,  p.  245)  for  the  estimation  of  item  difficulties  in  the  logistic 
Rasch  model. 

Fischer  also  suggests  a solution:  the  sequence  of  differences  be- 
tween values  obtained  at  successive  cycles  can  be  approximated  by  a geo- 
metric progression.  From  three  successive  values  one  may  estimate  the 
ratio  of  two  successive  differences,  and  use  this  ratio  to  predict  the 


"asymptotic"  value.  Consider  10*6^  in  Table  1 at  cycles  4,  5,  and  6: 

the  differences  are  20  and  16,  or  more  accurately,  20.03  and  15.50,  with 

a ratio  p ■ 15.50/20.03  - 0.774.  The  predicted  asymptotic  value  is  now 

3206.94  + 15.50  p/(l  - p)  * 3259.78.  Although  this  is  still  far  from  the 

actual  value  of  3433  given  in  the  last  line  of  the  table,  the  undisturbed 

iteration  process  needs  eight  more  steps  from  cycle  6 until  lO^B^  reaches 

3260.  Similar  calculations  for  the  other  parameters  lead  to  predictions  of 

2007,  2083,  2082,  482,  and  4803  for  the  other  columns.  A glance  at  Table  1 

4 5 

shows  that  this  leads  us  astray  for  10  and  10  8^,  for  which  the  direc- 
tion of  change  is  reversed  at  the  eighth  cycle;  it  slightly  overestimates 
4 

the  change  in  10  and  underestimates  the  changes  in  the  other  parameters. 

Not  knowr.ng  the  final  values,  one  could  continue  a few  more  cycles 
from  the  values  obtained  by  a "leap"  after  the  sixth  step.  Such  an  alter- 
nation of  cycles  and  leaps,  with  a few  refinements  to  be  described  next, 
produces  the  estimation  process  described  in  Table  2.  It  is  seen  that 
the  leap  after  cycle  14  already  produces  a higher  criterion  * log  posterior 
density  value  than  obtained  in  50  Iteration  steps  of  the  undisturbed  process. 
This  holds  for  both  runs  described  in  Table  2 (see  appendix) . A discussion 
of  the  difference  between  those  runs  is  postponed  until  section  5, 

Some  qualifications  and  refinements  art  necessary  for  the  satisfactory 
behavior  of  an  alternation  of  leaps  and  iterations.  We  shall  illustrate  this 
by  considering  the  parameter  B^  *n  Table  3 (see  appendix)  (column  marked  B^^ 
for  third  predictor,  row  2 in  each  block  for  second  group),  for  cycles  4,  5 
and  6.  The  successive  differences  are  -6.02  and  -8.95.  The  geometric 
ratio  would  be  larger  than  unity,  but  it  is  certainly  unsatisfactory  to 
predict  that  the  whole  sequence  of  successive  6^  values  diverges  to  -*>. 


Another  problem  is  met  for  a^,  where  the  successive  differences  are 

seen  from  Table  3 to  be  +1.82  and  -.20.  Does  this  Imply  a next  difference 

of  +.02  and  then  a still  smaller  negative  one,  as  would  follow  in  our 

geometric  series  model?  Although  this  might  be  true,  it  is  far  more 

plausible  that  we  have  hit  upon  a point  in  the  undisturbed  iteration 

process  where  an  initial  increase  is  just  changing  into  a stable  decrease 

4 

(cf.  the  column  for  10  in  Table  1). 

A third  problem  is  the  instability  of  the  estimated  ratio  p,  because 
of  cross-inf luences  from  other  parameter  estimates  and  because  of  systematic 
deviations  from  the  geometric  model.  When  p is  estimated  from  three  cycles 
ending  at  cycle  number  5,  6,  7,  8,  9,  10,  respectively,  for  the  parameter 
from  a version  of  Table  1 with  more  decimals,  one  obtains  .950,  .774, 
.726,  .814,  .811,  .848.  Such  behavior  is  certainly  not  atypical  and  it 
shows  how  rough  an  extrapolation  from  three  cycles  can  be. 

For  a solution  of  these  problems,  three  strategies  were  considered: 

a.  consider  more  than  three  cycles  and  use  an  extrapolation  method 
geared  to  what  was  observed; 

b.  consider  more  than  three  cycles  and  make  extrapolations  only  for 
those  parameter  values  for  which  conformity  to  the  geometric  series 
model  was  observed; 

c.  use  the  ratio  averaged  across  groups  in  the  extrapolation. 

There  is  empirical  evidence  that  each  of  the  strategies  improves  the 
prediction  of  the  final  parameter  values,  compared  to  the  simple  leaps  des- 
cribed before.  Although  the  data  used  in  section  5 in  our  benchmark  examples 
contained  groups  of  rather  different  sample  size  and  regression  parameter 
values  alongside  with  more  homogeneous  subgroups,  there  Is  a substantial 
similarity  between  groups  when  one  considers  the  changes  in  parameter 
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values  from  cycle  to  cycle.  The  dominant  behavior,  to  our  surprise,  was 
not  a general  regression  to  the  mean  (across  groups)  of  the  least  squares 
solutions,  but  a shif  vhich  some  regression  parameters  (B^»  &2’  and 
in  the  "10HETERO"  data  of  Tables  1-3)  grew  larger  at  the  expense  of  others 

this  data  set).  Although  the  risk  of  capitalizing  on  the  peculiarities 
of  some  data  sets  should  not  be  lightly  dismissed,  the  evidence  built  up  at 
this  stage  seems  to  indicate  that  a successful  extrapolation  of  the  itera- 
tion process  is  indeed  possible. 

All  three  strategies  (a),  (b) , and  (c),  however,  have  a major  draw- 
back when  considered  in  the  light  of  the  goal:  inclusion  of  m-group  re- 
gression into  the  CADA  Monitor.  They  all  involve  a considerable  amount 
of  "bookkeeping"  within  the  program,  which  might  go  as  far  as  doubling 
the  memory  space  and  execution  time  required  per  iteration  step.  For 
CPU  time,  this  is  easily  gained  back  because  of  the  drastic  reduction  in 
the  number  of  cycles  required,  but  for  use  on  medium  sized  computers  the 
core  limitations  impose  strong  restrictions  on  the  number  of  values  that 
can  be  kept  available  during  the  iteration  process.  Moreover,  there  is 
room  for  doubt  about  the  stability  of  an  iteration  cycle  that  might  start 
with  some  parameter  values  very  close  to  the  optimal  ones  and  others 
still  very  far  away  from  it. 

It  was  decided  for  this  reason  that  "leaps"  would  be  programmed  for 
all  parameters  and  groups  simultaneously,  and  would  be  based  on  the  indi- 
vidual geometric  ratios  calculated  from  the  past  three  values.  The  fac- 
tor p/(l  - p)  would  be  replaced  by  a fixed  number  SCH  for  cases  where  the 
past  two  differences  had  opposite  signs,  by  a fixed  number  DCN  for  cases 
where  the  last  difference  was  larger  than  its  predecessor  [or  so  little 

less  that  p/(l  - p)  would  exceed  DCN],  and  by  zero  if  the  last  difference 

—NDR 

meant  a relative  change  of  less  than  10  in  the  parameter  value.  The 


numbers  SCH,  DCN,  and  NDB  can  be  changed  by  the  user  of  the  program;  their 
default  values  have  been  chosen  as  20,  10,  and  4,  respectively,  on  the 
basis  of  experiments  described  in  section  5. 

That  section  also  describes  the  empirical  answer  to  the  question  of 
how  often  one  t hould  insert  leaps  between  the  normal  iteration  cycles. 

A first  leap  after  5 cycles  followed  by  leaps  after  each  4 cycles  seems 
to  produce  very  fast  convergence,  but  it  is  generally  true  that  many  other 
strategies  lead  to  essentially  the  same  criterion  values  in  only  a few 
extra  steps. 

As  an  occasional  "bad  leap"  could  produce  a decrease  of  the  criterion 
in  some  situations,  the  parameter  values  before  the  leap  are  stored,  and 
the  process  restarts  iteration  from  those  values  when  the  criterion  value 
after  the  leap  followed  by  one  iteration  is  less  than  the  criterion  value 
before  the  leap.  This  "bad  leap"  provision  should  be  distinguished  from 
the  "reduced  step"  provision  (E)  of  section  2,  which  applies  to  decreases 
of  the  criterion  occuring  at  other  stages  than  just  after  leaps. 

5.  Experiences  for  Four  Data  Sets 

At  the  time  of  the  author’s  short  visit  to  Iowa  City,  only  one  data 
set  was  directly  available.  It  consists  of  the  values  of  the  observed 
Grade  Point  Average  (predictand)  and  the  ACT  test  scores  English,  Mathe- 
matics, Social  Sciences,  and  Natural  Sciences  (four  predictors)  for  the 
pupils  of  22  junior  colleges,  out  of  which  a 25  per  cent  sample  of  the 
1968  data  was  obtained  as  described  in  Novick,  Jackson,  Thayer,  and  Cole 
(1971,  1972).  This  "25  per  cent  1968"  data  set  for  all  22  colleges  and 
all  four  predictors  will  be  denoted  as  "22&4"  in  the  sequel.  It  leads  to 


modal  estimates  In  which  the  slopes  for  the  fourth  predictor  are  totally 
regressed. 

Out  of  these  22  colleges,  we  selected  the  ones  numbered  1,  2,  4,  9,  14, 
16,  17,  20,  21,  22  because  they  led  to  widely  differing  slope  values;  this 
data  set  will  be  called  "10HETERO"  for  obvious  reasons,  and  its  modal  solu- 
tion contains  no  parameters  with  total  regression  to  the  mean.  The  re- 
maining 12  colleges  form  the  data  set  "12H0M0,"  for  which  the  second  and 
fourth  predictor  show  total  regression.  Finally,  in  the  spirit  of  Jones 
and  Novick  (1972,  p.  11),  we  analyzed  all  22  schools  with  only  the  first 
and  third  predictor  left,  under  the  name  "22&2." 

For  each  of  the  four  data  sets,  50  iterations  were  carried  out  with  the 
original  BAYREG  program.  Although  some  common  elements  were  found  in  the 
iteration  behavior,  see  section  4,  the  data  sets  behaved  rather  differently 
in  other  respects.  A parallel  change  for  the  slopes  of  the  different  groups 
was  sometimes  observed  right  from  the  start,  sometimes  only  after  some 
four  to  six  iterations;  sometimes  such  changes  occurred  for  all  slopes, 
sometimes  only  for  a few.  Convergence  was  disappointingly  slow,  especially 
for  the  full  "22&4"  data  set  and  "lOHETERO"  but  also  for  "22&2."  There  are 
enough  differences  in  the  behavior  during  iterations  to  use  the  four  data 
sets  as  trial  examples  for  improved  convergence,  pending  further  experi- 
ments on  new  data  sets. 

The  parameters  guiding  the  leap  process  were  tried  out  in  various 
combinations.  The  main  conclusion  is  that  the  number  of  iterations,  and 
therefore  the  computer'  time,  required  to  reach  a point  at  which  the 
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criterion  remains  constant  in  five  digits,  is  reduced  to  one  third,  some- 
times even  less,  by  the  introduction  of  leaps.  Many  parameter  choices 
including  leaps  after  every  fourth  or  fifth  cycle  are  far  better  than 
the  undisturbed  process.  The  first  leap  seems  to  be  most  effective  when 
taken  after  the  fifth  or  sixth  cycle. 

Although  the  exact  leaps  + iterations  behavior  varies  for  different 
choices  of  the  leap-governing  parameters,  many  such  choices  lead  to  almost 
the  same  strong  gain  in  speed.  Even  if  the  final  criterion  value  at  which 
the  process  stops,  for  one  of  the  reasons  given  in  section  2 under  (F) , is 
not  quite  the  same,  the  modal  estimates  tend  to  vary  only  slightly  from 
one  run  to  the  next  run  with  different  NDH,  NDB,  NCI,  NCF , SCH,  or  DCN. 

A typical  illustration  is  given  in  Table  4 (see  appendix),  where  it  is  seen 
that  the  model  estimates  are  almost  the  same  for  all  lines  except  the  last 
ones  which  pertain  to  the  undisturbed  iteration  process.  Sometimes,  however, 
almost  the  same  regression  coefficients  may  still  produce  somewhat  different 
outcome  of  a cross  validation  study,  as  will  be  discussed  in  section  6. 

Since  the  main  text  of  this  report  was  written,  the  author  obtained 
access  to  the  BOVO  MAVO  72/73  data  from  Jansen  (1977)  with  twelve  groups 

and  two  predictors,  of  which  one  has  a negative  ASY  estimate  for  t . The 

P 

undisturbed  iteration  process  has  a log  posterior  density  of  -1056.9005 
at  cycle  4,  -1056.5288  at  cycle  5,  -1056.4985  at  cycle  6,  and  remains 
constant  at  -1056.4916  from  cycle  15  onwards.  A leap  process  with  all 
default  options  jumps  from  cycle  5 to  -1056.4915  at  cycle  6.  A leap  from 
cycle  4 even  jumps  to  -1056.4905  at  cycle  5.  Although  convergence  for 
these  data  was  already  fast,  leaps  again  reduce  compute  time  by  a factor 


between  2 and  3. 


27 


6.  The  Road  Ahead 

This  section  contains  a brief  discussion  of  some  points  on  which  future 
work  in  the  area  might  be  concentrated. 

1.  As  mentioned  in  conclusion  (6)  and  shown  in  Table  4,  many  somewhat 
different  parameter  sets  lead  to  very  nearly  the  same  value  of  the  log 
posterior  density.  In  most  cases  the  goal  will  be  good  prediction  in 
future  applications.  The  experiences  thus  far  seem  to  indicate  that  the 
use  of  two  similar  sets  of  regression  coefficients,  both  close  to  the 
modal  ones,  produces  somewhat  unpredictable  results  at  cross-validation: 
the  quality  of  prediction  changes  sometimes  very  little,  sometimes  rather 
much.  Obviously  this  also  depends  on  the  operationalization  of  "quality 
of  prediction":  mean  squared  error,  absolute  error,  and  correlation 

can  be  expected  to  have  a more  continuous  behavior  than  zero-one  or 
threshold  loss  functions.  In  the  presence  of  outliers,  however,  it  has 
been  observed  for  some  data  sets  used  in  cross-validation  that  even  the 
former  three  criteria  may  react  rather  sharply  to  small  changes  in  the 
regression  equations.  The  average  increase  of  prediction  quality  men- 
tioned in  earlier  cross-validation  studies  is  indeed  an  average,  formed 
in  most  cases  from  a substantial  Improvement  in  some  groups  and  a de- 
terioration in  a few  others.  Moreover,  improvement  in  mean  squared 
error  sometimes  goes  with  decreasing  correlation  or  absolute  error, 
or  the  other  way  around. 

2.  In  the  light  of  recent  work  by  Smith  (1977),  it  may  be  time  to  repeat 


the  question  of  whether  modal  estimates  are  Indeed  the  best  end  product 
of  an  m-group  regression  analysis.  Although  the  answer  may  depend  heavily 
on  the  particular  circumstances  and  the  careful  specification  of  prior  in- 
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formation,  it  may  be  worthwhile  to  explore  in  more  detail  whether  there 
are  other  estimates  which  can  be  obtained  without  too  much  trouble,  and 
whether  modal  estimates  lead  to  better  predictions  than  those  others. 

3.  It  remains  to  be  investigated  whether  the  models  of  Shigemasu  (1976) 
can  be  incorporated  in  the  general  m-group  regression  program.  Both 
the  use  of  equal-slopes-unequal-intercepts  and  the  relaxation  of  the 
exchangeability  assumption  could  widen  the  scope  of  the  method. 

4.  Most  papers  on  m-group  regression  state  that  the  matrix  I occurring  in 

the  I.indley  equations  should  be  taken  as  a diagonal  matrix  with  three 

times  the  a priori  most  probable  values  of  the  variances  t0  and  t„. 

p ph 

(h  = 1,  2,  ...,£)  on  the  diagonal.  As  discussed  in  section  2,  the 
computer  programs  BPREP  and  BAYREG  take  as  "most  probable  values"  the 
variances  of  the  ASY  (essentially  model  II)  estimates,  after  revision 
for  possible  negative  variance  estimates.  These  estimates,  however, 
are  calculated  from  the  data:  their  use  means,  strictly  speaking, 
that  the  posterior  distribution  is  no  longer  the  product  of  prior  and 
likelihood.  The  point  is  unimportant  as  long  as  the  choice  of  the  r 
estimates  has  very  little  influence  on  the  end  result.  Lindley  (1969), 
after  explaining  that  such  estimates  are  necessary  to  prevent  diver- 
gence, comments  (page  8) , "If  these  terms  play  an  important  role  in 
the  calculations  described  below,  then  this  indicates  that  the  sample 
is  providing  information  comparable  in  magni-ude  with  that  available 
a priori:  that  is,  very  little." 

Our  problem  is  just  this:  in  the  five  data  sets  studied,  the  choice  of 
these  terms  turns  out  to  have  a strong  influence  on  the  iterative  process 
leading  to  the  modal  estimates  and  on  the  final  maximum  of  the  log  pos- 
terior density,  and  a weaker,  but  non-negligible,  influence  on  the  modal 
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estimates  of  the  various  regression  parameters.  It  will  be  investigated 
in  the  next  months  whether  this  is  also  true  for  other  data  sets,  and 
whether  a choice  0 < v'  < 1 for  the  degrees  of  freedom  of  the  Wishart 
distribution  of  the  inverse  of  the  regression  parameters  covariance 
matrix,  diminishing  the  influence  of  the  prior  information,  could  solve 
the  problem. 

5.  A further  simplification  of  the  program  would  be  possible  if  a geometric 
ratio  p derived  from  the  past  three  values  of  0^,  the  parameter  averaged 
across  groups,  could  be  used.  It  may  take  more  than  four  to  six  cycles 


before  the  ratios  for  different  groups  are  sufficiently  alike  to  make 
such  averaging  productive.  It  remains  to  be  seen  whether  the  p from 
8*,  common  to  all  groups  but  different  per  predictor  (or  intercept  or 
residual  variance),  should  then  be  applied  to  the  difference  of  0^ 

(this  would  do  away  with  most  of  the  arrays  of  size  25  by  5 now  in  the 
program)  or  to  the  individual  differences  per  predictor  and  per  group. 

A still  further  step  in  the  same  direction  would  be  the  replacement  of 
the  iteration  process,  after  a number  of  cycles,  by  a step  of  the  form 

“i.t+l  = ai,t  + Ca;  3hi,t+l  = 6hi,t  + Ch  (h  = lf  2’  ' * **  and 

t + c$ ’ the  constants  c would  be  determined  so  as 

to  maximize  the  log  posterior  density.  Such  a step,  uniform  across 
groups,  could  then  be  followed  by  a new  set  of  iterations  from  its  end- 
point. Research  along  these  lines  is  now  in  progress.  Its  feasibility 
is  illustrated  by  Table  5 (see  appendix),  which  gives  the  differences 
between  the  values  at  cycle  6 and  the  model  estimates  for  the  two  data 
sets.  Especially  for  0^,  B^»  and  8^  with  "lOHETERO,"  and  8j  and  0^,  with 

"22&2,"  where  the  most  drastic  changes  occur,  the  differences  do  not 
differ  too  much  across  groups. 


' 
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6.  As  stated  in  conclusion  (7)  of  section  2,  the  program  may  produce  some 
undesirable  results  when  faced  with  unusual  input  data,  e.g., 

- when  some  predictor (s)  or  the  predictand  are  scaled  with  an  extremely 
large  or  extremely  small  scale  unit, 

- when  some  predictor(s)  or  the  predictand  have  a zero  scale  point  very 
far  from  the  actual  values, 

- when  all  or  some  groups,  possibly  the  small  ones,  produce  an  (almost) 
perfect  fit  to  the  least  squares  solution. 

Rescaling  is  advised  in  the  first  and  second  case;  in  the  CADA  Monitor 
version  this  burden  could  be  taken  away  from  the  user  by  following  a 
suggestion  of  Dr.  Charles  Lewis:  let  the  program  rescale  all  predic- 
tors to  mean  square  deviation  1 around  the  ideal  points,  and  rescale 
the  predictand  to  zero  mean  and  unit  variance.  In  the  third  case,  col- 
lection of  more  data  or  deletion  of  some  predictors  may  reduce  the  dan- 
ger of  overfitting;  a careful  comparison  of  the  least  squares  solution 
for  the  residual  variances  to  the  minimum  PHIM1N  is  recommended. 

Ac  knowl edgmen  t 

I would  like  to  express  my  gratitude  for  the  continuous  encouragement 
and  assistance  of  Dr.  Melvin  R.  Novick,  Mr.  Gerald  L.  Isaacs,  and 
Dr.  Charles  Lewis.  I also  thank  Anne  Boomsma  for  detecting  some 
flaws  in  the  presentation,  and  Dr,  Margo  Jansen  for  giving  access  to 
her  data. 


r ” 


32 


References  (continued) 

Novick,  M.  R. , Jackson,  P.  H.,  Thayer,  D.  T.,  & Cole,  N.  S.  Estimating 

multiple  regressions  in  m-groups:  A cross-validation  study.  British 
Journal  of  Mathematical  and  Statistical  Psychology.  1972,  25,  33-50. 

Shlgemasu,  K.  Development  and  validation  of  a simplified  m-group  regres- 
sion model.  Journal  of  Educational  Statistics,  1976,  1,  157-180. 

Smith,  A.  F.  M.  Random  effects  models  involving  nonnormal  distributions. 
Invited  paper  presented  at  the  European  Meeting  of  Statisticians, 
Louvain,  August  1977. 


Table  1 

Parameter  values  for  the  first  group,  and  criterion-log  posterior  density. 


during 

the  iteration 

i process 

for  the 

undisturbed 

iterations 

for  the 

"10HETER0"  data  set 

described 

in  section  5. 

Cycle 

no. 

Criterion 

io4^ 

10\i 

10\l 

10Si 

10\i 

4 

10  $ 

i 

79.3542 

1993 

3175 

1687 

2542 

890 

4948 

2 

85.0815 

1949 

3154 

1916 

2344 

707 

4757 

3 

85.5006 

1968 

3150 

2001 

2320 

587 

4767 

4 

85.5679 

1987 

3171 

2044 

2308 

530 

4779 

5 

85.5943 

1998 

3191 

2064 

2298 

504 

4788 

6 

85.6120 

2003 

3207 

2074 

2288 

492 

4793 

7 

85.6267 

2005 

3219 

2079 

2278 

488 

4797 

8 

85.6400 

2005 

3228 

2081 

2268 

487 

4799 

9 

85.6523 

2004 

3235 

2082 

2258 

489 

4800 

10 

85.6639 

2003 

3242 

2083 

2249 

491 

4801 

15 

85.7141 

1994 

3266 

2085 

2206 

509 

4802 

20 

85.7535 

1985 

3285 

2086 

2167 

528 

4802 

25 

85.7846 

1977 

3302 

2087 

2133 

544 

4802 

30 

85.8090 

1969 

3318 

2088 

2103 

558 

4802 

35 

85.8283 

1963 

3331 

2088 

2076 

571 

4802 

40 

85.8434 

1957 

3343 

2089 

2052 

582 

4802 

45 

85.8554 

1952 

3354 

2090 

2031 

592 

4802 

50 

85.8648 

1948 

3363 

2090 

2012 

601 

4802 

55 

85.8722 

1944 

3371 

2091 

1995 

608 

4802 

60 

85.8781 

1941 

3379 

2091 

1980 

616 

4802 

65 

85.8827 

1938 

3385 

2091 

1967 

622 

4802 

70 

85.8863 

1935 

3391 

2092 

1956 

628 

4802 

75 

85.8892 

1932 

3396 

2092 

1945 

633 

4802 

80 

85.8915 

1930 

3401 

2092 

1936 

637 

4802 

85 

85.8932 

1928 

3405 

2093 

1928 

641 

4802 

90 

• ** 

85.8946 

1927 

3409 

2093 

1921 

644 

4802 

95 

85.8958 

1925 

3412 

2093 

1914 

647 

4802 

* 

Max 

85.8999 

1915 

3433 

2094 

1871 

668 

4802 

This  line  could  have  been  obtained,  in  principle,  by  taking  many  more  itera- 
tion cycles,  but  it  was  actually  obtained  at  the  37th  iteration  of  a leap 
process  (Table  2). 
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Table  2 

Parameter  values  for  the  first  group,  and  criterion  values  ‘"or  two  runs  of 


the  speeded  program  applied 

to  the 

"10HETER0" 

data  set  desc 

. ibed  in 

sectior 

for  NDB 

» 3,  NDH  - 

5,  NCI  » 

6,  NCF 

* A, 

DCN  - 

SCH  * 20,  VGT 

- .5,  NCY  » 25 

(first 

half)  and  NDB  = 4,  NDH  - 6, 

NCI 

- 6,  NCF  = 4,  DCN  » 

30,  SCH  * 

20, 

VGT  - . 

5,  NCY  » 50 

(below  the  solid 

line) . 

Cycle 
No. 

Criterion 

104a1 

l°56n 

lo582l 

10\l 

10\i 

4 

io%  x 

6 

85.6120 

2003 

3207 

2074 

2288 

492 

4793 

Leap 

- 

2007 

3260 

2083 

2082 

482 

4803 

7 

85.7391 

1981 

3289 

2095 

2160 

531 

4798 

10 

85.7789 

1978 

3298 

2087 

2139 

541 

4801 

Leap 

— 

1978 

3375 

2087 

2006 

602 

4801 

11 

85.8298 

1959 

3345 

2090 

2057 

582 

4802 

14 

85.8480 

1956 

3347 

2089 

2044 

586 

4802 

Leap 

- 

1956 

3347 

2089 

1957 

626 

4802 

15 

85.8690 

1940 

3369 

2092 

1991 

616 

4802 

18 

85.8772 

1940 

3376 

2091 

1982 

616 

4802 

Leap 

— 

1940 

3376 

2091 

1925 

630 

4802 

19 

85.8862 

1931 

3392 

2095 

1948 

634 

4802 

22 

85.8900 

1931 

3397 

2092 

1942 

635 

4802 

Leap 

- 

1931 

3397 

2092 

1942 

635 

4802 

23 

85.8913 

1930 

3399 

2093 

1937 

637 

4802 

25 

85.8921 

1929 

3402 

2093 

1933 

639 

4802 

6 

85.6120 

2003 

3207 

2074 

2288 

492 

4793 

Leap 

- 

2007 

3260 

2083 

1980 

482 

4803 

7 

85.7515 

1968 

3309 

2101 

2114 

555 

4803 

10 

85.8135 

1967 

3319 

2089 

2096 

562 

4802 

Leap 

— 

1944 

3439 

2088 

1929 

630 

4802 

11 

85.8503 

1953 

3369 

2090 

2021 

596 

4802 

14 

85.8655 

1948 

3365 

2090 

2010 

601 

4802 

Leap 

- 

1940 

3377 

2099 

1904 

666 

4802 

15 

85.8804 

1931 

3390 

2093 

1952 

636 

4802 

18 

85.8889 

1932 

3395 

2092 

1946 

633 

4802 

Leap 

— 

1932 

3441 

2092 

1887 

637 

4802 

19 

85.8934 

1927 

3412 

2094 

1920 

644 

4802 

22 

85.8955 

1926 

3412 

2093 

1915 

646 

4802 

Leap 

- 

1923 

3412 

2093 

1878 

652 

4802 

23 

85.8971 

1919 

3420 

2094 

1895 

659 

4802 

37 

85.8999 

1915 

3433 

2094 

1871 

668 

4802 

Table  3 

Fragment  of  the  computer  output  describing  the  "leap"  based  on  the  values  at 
cycles  4,  5 and  6 for  the  "10HETERO"  data,  see  text.  After  a first  line 
giving  cycle  number  and  criterion  = log  posterior  density,  there  Is  a line 
for  each  group  giving  Its  parameter  values  at  this  cycle. 


cycle  group 

i a - 

4 .05567^19£*O2 

4 1 • 19B722E*00 

4 2 .210253€>OO 

4 3 ,300992E*00 

^ . 4 • 10931 0E*00 

4 5 ,241519E*00 

4 6 • lb8049E ♦00 

4 7 ,148270t*00 


•31 7141E-01 

•23356SE-01 
•167197E-01 
• <.4403bE-01 
•366613E-01 
•475703E-01 
•299882E-Q1 


4 8 

4 9 
' 4 10 

5 

5 1 
5 2 
5 3 
5 4 

5 

S 6 
5 7 
S 0 
S 9 

5 10 

6 

6 1 
6 2 
6 3 
6 4 
6 5 

••  6 
6 7 
6 8 
6 9 
6 10 


•424079E*00 
•174106E-01 
•909752E-01 
• 85594 306E* 02 
•199776E*00 
•219320£*00 

• 30 1490E  *00' 
• 189980E*00 
•242 1 1 7E*00 
• 15b808E*00 
•148*52E*00 

• 424476E  *00 
• ) 81830E-0 1 
•91S989E-01 

•85612C01E*02 
•20027tE*00 
•219708E*OO 
•301645E*00 
•190333E*00 
•24?367E*00 
• 159203E*00 
•140432E*OO 
•4246l4£*00 
•185756E-01 
•918755E-01 


•226S37E-01 

•463704E-01 

•364153E-01 

•319144E-01 
• 235.883E-01 
•170231E-01 
•446012E-01 
• 368874E-0 1 
•477688E-01 
•302016E-01 
•229538E-01 
•465688E-01 
•366290E-01 

•320694E-01 
•237567E-01 
• ) 72209E-0 1 
•447654E-0 1 
• 370557E-0 1 
•479215E-01 
•303495E-01 
•231623E-01 
•467186E-01 
•367861E-01 


AFTER  LEAP  FROM  CYCLE  6 


6 1 
6 2 
6 3 

6 4 

6 5 

6 6 
^ 7 

6 0 


.2OO720E*OO 
.220 153E*00 
•301645E*00 
• 190722£*00 
• 242547£>00 
•159632€*00 
•148432E*00 
•424614E*00 


.325978E-01 

•242042E-01 

•175909E-01 

•455724E-01 

•378460E-01 

•404315E-O1 

•306840E-01 

•236360E-01 


6 9 .109017E-O1  .471812E-01 
6 10  .920960E-01  .372226E-01 


3<. 


.2044 1 0£"-0 1 
. 196J12E-01 
•324606E-01 
. J28801E-01 
. 338  794E-0  1 
•346871E-01 
•249V0bE-01 

• 1 B0bQ9E-0  1 
. 351 141E-01 
•2927bbE-01 

•206426E-01 
.1991 13E-01 

• 32554 1 E-0 1 
.331 198E-01 
• 340232E-0 1 
.348564E-01 
•250832E-01 
. 182195E-01 

• 352922E-0  1 
•2941 73E-0 1 

•207384E-01 
. 197500E-01 
• 32596QE-0 1 

• 33225 1 E-0 1 
.340872E-01 
. 349309E-0  1 
•251281 E-0 1 
. 182934E-01 
•353748E-01 
•29483VE-01 


.208252E-01 

• 1 9 78S9E-0 1 
• 32b299E-0 1 

• 3330  7 7E-0 1 
•341 383E-0 1 
• J49B93E-01 

• 25 1 7 04E-0 1 
• 183580E-01 
• 354462E-0 1 
•295440E-01 


•230846E-01 
•225506E-0  1 
•221822E-01 
•23b572E-01 
.231 760E-01 
•234329E-01 
• 22250 1 E-0 1 
•222705E-0 1 
•23b781E-01 
•2J2540E-01 


•529719E-02 
-. 1920 14E-01 
,b48 Jb  7E-03 
•26J572E-0 1 
.845J43E-02 
. ! 0b5 1 3E-0 1 
-.880b45E-02 
•348202E-02 
. 7J9955E-02 
•957513E-03 


*1 

•477939E*00 
,4b 1 1 79E  *00 
•4b7857E*00 
•466738E*00 
•494b29E*00 
•4718b4E*00 
•4b0792E*00 

• 46b9  7bE*00 
•■481816E*00 
•471437E*00 

• 478  776E400 
•461952E*00 
•4b8b23E«00 
•467537E*00 
•495387E»00 
•4725b0E»00 
-46l403£*00 
•bb7695E*00 
•482508E«00 
•4721 07E»00 

•479315E*00 
.4b2452E*00 
•4b91 1 1£«00 
•4b8048E*00 
•495879E»00 
,47300bE»00 
•4b 1 789E*00 
•468152E+00 
•482953E*00 
•472538E*00 

.480293E400 

,4b3370E*00 

•4b9975E*00 

,468949E*00 

• 495079£-»OO 
•473006E+00 
•46 1 789E*00 

• 4681 52E  *00 
•482953E*00 

• 4725^E«00 


•229780E-0 1 .503686E-02 
•224904E-0 1 -. 1 9353 3E-0 1 
•22131 3E-0 1 .455729E-03 

• 2349b  7E-0 1 .2S9S23E-01 
•230579E-0 1 .827359E-02 
•232778E-01  .104335E-01 
•22J496E-01-.893549E-02 
•222256E-01  .322406E-02 
•235120E-0 1 • 721 302E-02 
•231254E-01  . 786180E-03 

•228754E-01  .491976E-02 
•224009E-01-.194162E-01 
•220454E-01  .365874E-03 
•233780E-01  ,25754lE-0l 
•229523E-C 1 .819379E-02 
•231612E-01  • 1 03308E-0 1 
•220507E-01-. 898840 E -02 
•22141 7E-01  . 31 09b8E-02 
•233923E-0 1 .712385E-02 

• 230167E-01  • 70<*763E-03 


•208222E-01  .482403E-02 
•206120E-01-, 194606E-01 
•203269E-0 1 .287323E-03 
•2304 1 7E-0 1 .255639E-01 
•220482E-01  *8130 12E-02 
•228098E-01  .102390E-01 
•200718E-0 1 -.90c61 6E-02 
•204640E-01  .301857E-02 
•230829E-01  .704218E-02 
•224268E-01  .631044E-03 


Results  of  several  runs  for  the  "10HETERO"  data,  first  group.  The  last 
column  gives  the  cycle  number  at  which  the  result  was  obtained,  followed 
by  "E'  if  Iteration  ended  there  and  by  "I"  If  it  is  an  intermediate  result. 

4 5 5 5 5 4 
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Table  5 


Increments,  per  group  of  the  "10HETERO"  and  "22&2"  data  sets,  between 
parameter  values  at  cycle  6 and  modal  estimates  (cf.  Table  2).  Note 
similarity  of  increments  across  groups  [see  (5)  of  section  6J. 


"1 OHETERO"  Data  Set 


lO4^ 

10\i 

l0\i 

t 

CY=6 

modal 

incr . 

CY=6 

modal 

incr. 

CY=6 

modal 

incr . 

1 

2003 

1915 

-88 

3207 

3433 

226 

2074 

2094 

20 

2 

2198 

2110 

-88 

2376 

2633 

257 

1975 

1965 

-10 

3 

3016 

2906 

-110 

1722 

1962 

240 

3260 

3262 

2 

4 

1903 

1847 

-56 

4477 

4716 

239 

3322 

3369 

47 

5 

2424 

2324 

-100 

3706 

3970 

264 

3409 

3443 

34 

6 

1592 

1524 

-68 

4792 

5031 

239 

3493 

3518 

25 

7 

1484 

1349 

-135 

3035 

3257 

222 

2513 

2545 

32 

8 

4246 

4174 

-72 

2316 

2561 

245 

1829 

1813 

-16 

9 

1858 

1283 

-575 

4672 

4899 

227 

3537 

3576 

29 

10 

916 

829 

-87 

3679 

3905 

226 

2948 

2998 

50 

(continued) 

10\i 

10\i 

4 

10  \ 

i 

CY-6 

modal 

incr. 

CY*6 

modal 

incr . 

CY=6 

modal 

incr. 

1 

2288 

1871 

-417 

492 

668 

176 

4793 

4802 

9 

2 

2240 

1827 

-413 

-1942 

-1788 

154 

4625 

4633 

8 

3 

2205 

1793 

-412 

37 

197 

160 

4691 

4691 

4 

4 

2338 

1915 

-423 

2575 

2726 

151 

4680 

4687 

7 

5 

2295 

1879 

-416 

819 

984 

165 

4959 

4967 

8 

6 

2316 

1897 

-419 

1033 

1158 

125 

4730 

4734 

4 

7 

2205 

1797 

-408 

-899 

-748 

151 

4618 

4616 

-2 

8 

2214 

1801 

-413 

311 

482 

171 

4682 

4685 

3 

9 

2339 

1921 

-416 

712 

835 

123 

4830 

4836 

6 

10 

2302 

1887 

-415 

70 

191 

121 

4725 

4731 

6 

Table  5 (con't.) 
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"22&2"  Data  Set 


i 

CY-6 

4 

10  °i 

modal 

lncr . 

CY-6 

i°\i 

modal 

liter. 

CY-6 

i°5b2i 

modal 

incr . 

CY-6 

10%. 

modal 

incr. 

1 

624 

554 

-70 

3996 

4309 

313 

3268 

2910 

-358 

4753 

4773 

20 

2 

1128 

1076 

-52 

3534 

3874 

340 

3136 

2777 

-359 

4614 

4633 

19 

3 

2181 

2115 

-66 

4643 

4986 

343 

3244 

2884 

-360 

4720 

4737 

17 

4 

2385 

2296 

-89 

3298 

3629 

331 

3047 

2692 

-355 

4727 

4745 

18 

5 

1287 

1210 

-77 

4333 

4668 

335 

3210 

2859 

-351 

4636 

4651 

15 

6 

-1852 

-1895 

-43 

3646 

3942 

298 

3246 

2892 

-354 

4798 

4818 

20 

7 

1114 

1063 

-51 

4454 

4784 

330 

3283 

2918 

-365 

4655 

4673 

18 

8 

1728 

1659 

-69 

3944 

4285 

341 

3251 

2887 

-364 

4661 

4682 

21 

9 

635 

597 

-38 

4779 

5111 

332 

3386 

3022 

-364 

4678 

4698 

20 

10 

4253 

4111 

-142 

4398 

4758 

360 

3202 

2835 

-367 

4686 

4705 

19 

11 

5821 

5740 

-81 

4508 

4888 

380 

3159 

2788 

-371 

4695 

4711 

16 

12 

5712 

5590 

-122 

4367 

4760 

384 

3126 

2759 

-367 

4587 

4603 

16 

13 

983 

888 

-95 

3713 

4043 

330 

3155 

2798 

-357 

4676 

4692 

16 

14 

1318 

1242 

-76 

4603 

4940 

337 

3342 

2983 

-359 

4976 

4995 

19 

15 

1227 

1163 

-64 

4175 

4516 

341 

3244 

2886 

-358 

4634 

4651 

17 

16 

739 

713 

-26 

5103 

5430 

327 

3338 

2977 

-361 

4755 

4770 

15 

17 

142 

20 

-122 

3728 

4026 

298 

2965 

2631 

-334 

4628 

4636 

8 

18 

3387 

3258 

-129 

4172 

4531 

359 

3175 

2817 

-358 

4664 

4681 

17 

19 

5375 

5268 

-107 

4622 

5005 

383 

3224 

2956 

-368 

4666 

4684 

18 

20 

3994 

3924 

-70 

4224 

4594 

370 

3115 

2758 

-357 

4647 

4661 

14 

21 

-1763 

-1795 

-32 

4543 

4844 

301 

3379 

3028 

-351 

4854 

4871 

17 

22 

-606 

-668 

-62 

4161 

4470 

309 

3299 

2948 

-351 

4715 

4731 
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University  Memo 


To:  All  of  University  of  Iowa  Students  and  Employees 

From:  University  Business  Office 

Any  charges  made  to  your  University  Account  will  be  subject  to  the  following  terms: 

Agreement  and  Disclosure  Statement 


Participation  in  the  University’s  Open  End  Credit 
Plan  is  on  the  following  terms  and  conditions: 

1 . The  University,  from  time  to  time  and  in  its  sole  dis- 
cretion, may  permit  purchase  of  goods  or  servic' .-.  » i defer 
payment  thereof. 

2.  Payment  shall  be  made  as  follows:  On  or  before  fif- 
teen days  after  the  first  of  each  month,  a minimum  periodic 
payment  at  the  University  (by  mail  or  otherwise)  of  the 
New  Balance  shown  on  the  periodic  statement  for  that 
billing  is  required. 

3.  The  University  may,  at  any  time  and  in  its  sole  dis- 
cretion, terminate  this  Plan  or  revoke  any  or  all  privileges 
under  the  Plan.  In  any  event,  the  Plan  will  terminate  auto- 
matically upon  ceasing  either  to  be  registered  as  a student 
or  employed  by  the  University. 

4.  Obligations  shall  be  in  default  if: 

a)  any  payment  required  by  this  agreement  is  past  due 
more  than  ten  days,  or 

b)  Any  representation  made  to  induce  extension  of 
credit  under  this  Plan  proves  untrue,  any  covenant  of 
this  agreement  is  breached,  or  the  student  or  employee 
becomes  insolvent  or  bankrupt  so  as  to  materially 
impair  the  prospect  of  paying  obligations  under  this 
Plan. 

In  the  event  of  a default  (and  after  the  University  has  given 
any  notices  and  observed  any  waiting  period  which  may  be 
required  by  law)  the  entire  balance  under  this  Plan  shall  be- 
come due  and  payable  at  the  option  of  the  University. 

5.  Failure  to  pay  the  monthly  periodic  payment  when 
due  may  subject  a student  to  cancellation  of  registration. 
There  is  a $ 1 0.00  charge  for  the  first  reinstatement  of  regis- 
tration each  semester  or  session.  Reinstatement  is  usually  re- 
fused a second  time. 


In  Case  of  Errors  or  Inquiries  About  Your  Bill 

The  Federal  Truth  in  Lending  Act  requires  prompt 
correction  of  billing  mistakes. 

1.  If  you  want  to  preserve  your  rights  under  the  Act, 
here’s  what  to  do  if  you  think  your  bill  is  wrong  or  if  you 
need  more  information  about  an  item  on  your  bill: 

a.  Do  not  write  on  the  bill.  On  a separate  sheet  of  paper 
write  ( Alternate : Write  on  the  bill  or  other  sheet  of  paper] 
(you  may  telephone  your  inquiry  but  doing  so  will  not 
preserve  your  rights  under  this  law ) the  following: 

i.  Your  name  and  account  number  (if  any). 

ii.  A description  of  the  error  and  an  explanation  (to 
the  extent  you  can  explain)  why  you  believe  it  is  an 
error. 

If  you  only  need  more  information,  explain  the  item 
you  are  not  sure  about  and,  if  you  wish,  ask  for  evidence 
of  the  charge  such  as  a copy  of  the  charge  slip.  Do  not 
send  in  your  copy  of  a sales  slip  or  other  document  un- 
less you  have  a duplicate  copy  for  your  records. 

iii.  The  dollar  amount  of  the  suspected  error. 

iv.  Any  other  information  (such  as  your  address) 
which  you  think  will  help  the  creditor  to  identify  you 
or  the  reason  for  your  complaint  or  inquiry. 

b.  Send  your  billing  error  notice  to  the  address  on 

your  bill  which  is  listed  after  the  words:  “Send  Inquiries  To:.’ 


Mail  it  as  soon  as  you  can,  but  in  any  case,  early 
enough  to  reach  the  creditor  within  60  days  after  the 
bill  was  mailed  to  you.  If  you  have  authorized  your  bank 
to  automatically  pay  from  your  checking  or  savings  ac- 
count any  credit  card  bills  from  that  bank,  you  can  stop 
or  reverse  payment  on  any  amount  you  think  is  wrong 
by  mailing  your  notice  so  the  creditor  receives  it  within 
16  days  after  the  bill  was  sent  to  you.  However,  you  do 
not  have  to  meet  this  16-day  deadline  to  get  the  creditor 
to  investigate  your  billing  error  claim. 


2.  The  creditor  must  acknowledge  all  letters  pointing 
out  possible  errors  within  30  days  of  receipt,  unless  the 
creditor  is  able  to  correct  your  bill  during  that  30  days. 
Within  90  days  after  receiving  your  letter,  the  creditor  must 
either  correct  the  error  or  explain  why  the  creditor  believes 
the  bill  was  correct.  Once  the  creditor  has  explained  the  bill, 
the  creditor  has  no  further  obligation  to  you  even  though 
you  still  believe  that  there  is  an  error,  except  as  provided  in 
paragraph  S below. 

3.  After  the  creditor  has  been  notified,  neither  the 
creditor  nor  an  attorney  nor  a collection  agency  may  send 
you  collection  letters  or  take  other  collection  action  with 
respect  to  the  amount  in  dispute;  but  periodic  statements 
may  be  sent  to  you,  and  the  disputed  amount  can  be  applied 
against  your  credit  limit.  You  cannot  be  threatened  with 
damage  to  your  credit  rating  or  sued  for  the  amount  in  ques- 
tion, nor  can  the  disputed  amount  be  reported  to  a credit 
bureau  or  to  other  creditors  as  delinquent  until  the  creditor 
has  answered  your  inquiry.  However,  you  remain  obligated 
to  pay  the  parts  of  your  bill  not  in  dispute. 

4.  If  it  is  determined  that  the  creditor  has  made  a mis- 
take on  your  bill,  you  will  not  have  to  pay  any  finance 
charges  on  any  disputed  amount.  If  it  turns  out  that  the 
creditor  has  not  made  an  error,  you  may  have  to  pay  finance 
charges  on  the  amount  in  dispute,  and  you  will  have  to  make 
up  any  missed  minimum  or  required  payments  on  the 
disputed  amount.  Unless  you  have  agreed  that  your  bill  was 
correct,  the  creditor  must  send  you  a written  notification  of 
what  you  owe;  and  if  it  is  determined  that  the  creditor  did 
make  a mistake  in  billing  the  disputed  amount,  you  must 

be  given  the  time  to  pay  which  you  normally  are  given  to 
pay  undisputed  amounts  before  any  more  finance  charges 
or  late  payment  charges  on  the  disputed  amount  can  be 
charged  to  you. 


5.  If  the  creditor’s  explanation  does  not  satisfy  you 
and  you  notify  the  creditor  in  writing  within  10  days  after 
you  receive  his  explanation  that  you  still  refuse  to  pay  the 
disputed  amount,  the  creditor  may  report  you  to  credit 
bureaus  and  other  creditors  and  may  pursue  regular  col- 
lection procedures.  But  the  creditor  must  also  report  that 
you  think  you  do  not  owe  the  money,  and  the  creditor 
must  let  you  know  to  whom  such  reports  were  made.  Once 
the  matter  has  been  settled  between  you  and  the  creditor, 
the  creditor  must  notify  those  to  whom  the  creditor  report- 
ed you  as  delinquent  of  the  subsequent  resolution. 

6.  If  the  creditor  does  not  follow  these  rules,  the 
creditor  is  not  allowed  to  collect  the  first  $50  of  the  dis- 
puted amount  and  finance  charges,  even  if  the  bill  turns  out 
to  be  correct. 

7.  If  you  have  a problem  with  property  or  services 
purchased  with  a credit  card,  you  may  have  the  right  not 
to  pay  the  remaining  amount  due  on  them,  if  you  first  try 
in  good  faith  to  return  them  or  give  the  merchant  a chance 
to  correct  the  problem.  There  are  two  limitations  on  this 
right. 

a.  You  must  have  bought  them  in  your  home  state 
or  if  not  within  your  home  state  within  1 00  miles  of 
your  current  mailing  address;  and 

b.  The  purchase  price  must  have  been  more  than 
$50. 

However,  these  limitations  do  not  apply  if  the  merchant  is 
owned  or  operated  by  the  creditor,  or  if  the  creditor  mailed 
you  the  advertisement  for  the  property  or  services. 
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*«  DR.  JACK  ADAMS 

OFFICE  OF  NAVAL  RESEARCH  BRANCH 
22°  OLD  MARYLEBONE  ROAD 
LONDON,  NW,  15TH  ENGLAND 

1 Dr.  J3ck  R.  Borsting 
Provost  1 Academic  Dean 
U.S.  Naval  Postgraduate  School 
Monterey,  CA  939*10 

1 Dept,  of  the  Navy 
CHNAVMAT  (NMAT  03*1D) 

Washington,  DC  20350 

1 Chief  cf  Navel  Education  and 
Training  Support  )-(01A) 

Pensacola,  FL  32509 

1 Dr.  Charles  E.  Davis 
ONR  Branch  Office 
538  S.  Clark  Street 
Chicago,  IL  80505 

5 Dr.  Marshall  J.  Farr,  Director 

Personnel  \ Training  Research  Programs 
Office  of  Naval  Research  (Code  4'58) 
Arlington,  VA.  22217 

1 DR.  PAT  FEDERICO 

NAVY  PERSONNEL  R4D  CENTER 
SAN  DIEGO,  C«  92 152 

1 CDR  John  Ferguson,  MSC,  USN 

Naval  Medical  R4D  Command  (Code  UH) 
National  Naval  Medical  Center 
Bethcsda  , MD  2001 *1 

1 Dr.  John  Ford 

Navy  Personnel  P4D  Center 
San  Diego,  CA  92152 

1 Dr.  Eugene  E.  Gloye 

CNR  Branch  Office 
10?0  East  G^een  Street 
Pasadena,  CA  91101 


L 


1 CDR  Robert  S.  Kennedy 

Navel  Aerospace  Medical  and 
Research  Lab 
Box  29**07 

New  Orleans,  LA  70189 

1 Dr.  Norman  J.  Kerr 

Cnief  of  Naval  Technical  Training 
Naval  Air  Station  Memphis  (75) 
Millington,  TN  ?805*i 

1 Dr.  Leonard  Krocker 

Navy  Personnel  R4D  Center 
S.an  Diego,  CA  92152 

1 Dr . James  Lester 
CNR  Branch  Office 
*195  Siimrner  Street 
EosJtkSn,  MA  02210 

1 , ''Dr.  William  L.  Maloy 

Principal  Civilian  Advisor  for 
Education  and  Training 
Naval  Training  Command,  Code  00A 
Pensacola,  FL  32508 

1 Dr.  Jamns  McBride 
Code  301 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152 

2 Dr.  James  McGrath 

Navy  Personnel  R&D  Center 
Code  306 

San  Diego,  CA  92152 

1 DR.  WILLIAM  MONTAGUE 

NAVY  PERSONNEL  R&  D CENTER 
SAN  DIEGO,  CA  92152 

1 Commanding  Officer 
Naval  Health  Research 
Center 

Attn:  Library 

San  Diego,  CA  92152 
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1 CDR  PAUL  NELSON 

NAVAL  MEDICAL  R&  D COMMAND 
CODE  44 

NATIONAL  NAVAL  MEDICAL  CENTER 
PETHESDA , MD  20014 

1 DR.  RICHARD  J.  NIEHAUS 
CODE  301 

OFFICE  OF  CIVILIAN  PERSONNEL 
NAVY  DEPT 

WASHINGTON,  DC  20590 
1 Library 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152 

6 Commanding  Officer 

Naval  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

1 OFFICE  OF  CIVILIAN  PERSONNEL 
( CCDE  26) 

DEPT.  OF  THE  NAVY 
WASHINGTON,  DC  20390 

1 JOHN  OLSEN 

CHIEF  OF  NAVAL  EDUCATION  & 

TRAINING  SUPPORT 
PENSACOLA,  FL  32509 

1 Office  of  Naval  Research 
Cod-  200 

Arlington,  VA  22217 

1 Scientific  Director 

Office  of  Naval  Research 
Scientific  Liaison  Group/Tokyo 
American  Embassy 
APO  San  Francisco,  CA  96503 

1 SCIENTIFIC  ADVISOR  TO  THE  CHIEF 
OF  NAVAL  PERSONNEL 
NAVAL  PUREAU  OF  PERSONNEL  (PERS  OR) 
RM.  U4 Iff.  ARLINGTON  ANNEX. 


1 DR.  RICHARD  A.  POLLAK 

ACADEMIC  COMPUTING  CENTER 
U.S.  NAVAL  ACADEMY 
ANNAPOLIS,  MD  21402 

1 Mr.  Arnold  I.  Rubinstein 

Human  Resoureces  Program  Manager 
Naval  Material  Command  (0344) 

Room  1044,  Crystal  Plaza  #5 
Washington,  DC  20360 

1 A.  A.  SJOHOLM 

TECH.  SUPPORT,  CODE  201 
NAVY  PERSONNEL  R&  D CENTER 
SAN  DIEGO,  CA  92152 

1 Mr.  Robert  Smith 

Office  of  Chief  of  Naval  Operations 
OP-9S7E 

Washington,  DC  20350 

1 Dr.  Alfred  F.  Smode 

Training  Analysis  4 Evaluation  Group 
(TAEG) 

Dept,  of  the  Navy 
Orlando,  FL  32813 

1 DR.  H.M.  WEST  III 

DEPUTY  ADCNO  FOR  CIVILIAN  PLANNING 
AND  PROGRAMMING 
RM.  2625,  ARLINGTON  ANNEX 
WASHINGTON,  DC  20570 

1 DR.  MARTIN  F.  W1SK0FF 

NAVY  PERSONNEL  R&  D CENTER 
SAN  DIEGO,  CA  92152 


I 


1 HC  USAREUE  & 7th  A^my 
ODCSOPS 

USAAREUE  Director  of  GED 
APO  Now  York  0940' 

1 DR.  RALPH  CANTER 

U.S.  ARMY  RESEARCH  INSTITUTE 
0001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22”? 

1 DR.  RALPH  DUSSK 

U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22??^ 

1 Dr.  Milton  S.  Kat.z 

Individual  Training  4 Skill 
Evaluation  Technical  Area 
U.S.  Jray  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22 333 

1 Dr.  Harold  F.  CNMl,  Jr. 

ATTN:  PEfil-CK 

c001  EISENHOWER  AVENUE 

ALEXANDRIA,  VA  22??3 

1 DR.  JAMES  L.  RANEY 

U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA,  VA  22?”. 

1 Director,  Training  Development 
U.S.  Arny  Administration  Center 


1 Air  Force  Human  Resources  Lab 
AFHRL/PED 

Crooks  AFP,  TX  73235 

1 Air  University  Library 
AUL/LSE  75/4U3 
Maxwell  AFE,  AL  ?6112 

1 Dr.  Alfred  R.  Fregly 
AFGSR/NL,  Eldg.  410 
Polling  AFE,  DC  20??2 

1 Dr.  Ross  L.  Morgan  ( AFHRL/ASR) 
Wright  -Patterson  AFP 
Ohio  45433 

1 Personnel  Analysis  Division 
HQ  USAF/DPXXA 
Washington,  DC  20330 

1 Research  Branch 
AFMPC/DPMYP 

Randolph  AFP,  TX  73148 

1 Dr.  Marty  Rock way  ( AFHRL/TT ) 
Lowry  AFP 
Colorado  80230 

1 Major  Wayne  S.  Sellman 
Chief,  Personnel  Testing 
AFHPC/DPMYPT 
Randolph  AFE,  TX  78148 


ATTN:  Dr.  Sherrill 

Ft.  penjsmin  Harrison,  IN  46218 

1 Dr.  Joseph  Ward 

U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  223?? 

“L 


owa/novick  TUESDAY,  MAY  23,  197?  98:56:04-PDT 


PAGE  4 


Marines 


1 Director,  Office  of  Manpower 
HC,  Marine  Corps  (MPU) 

PCE , Pldg.  2009 
Quantico , VA  221 34 

1 DR.  A . L.  SLAFKOSKY 

SCIENTIFIC  ADVISOR  (CODE  RD- 
HQ,  U.S.  MARINE  CORPS 
WASHINGTON,  DC  20R80 


Other  DoD 


Utilization  1 Dr.  Stephen  Andriole 

ADVANCED  RESEARCH  PROJECTS  AGENCY 
1400  WILSON  BLVD. 

ARLINGTON,  VA  22209 

12  Defense  Documentation  Center 
1)  Cameron  Station,  Bldg.  5 

Alexandria,  VA  22314 
Attn:  TC 


1 Dr.  Dexter  Fletcher 

ADVANCED  RESEARCH  PROJECTS  AGENCY 
1400  WILSON  BLVD. 

ARLINGTON,  VA  22209 

1 Military  Assistant  for  Human  Resources 
Office  of  the  Director  of  Defense 
Research  4 Engineering 
Room  3D129,  the  Pentagon 
Washington,  DC  20301 

1 Director,  Research  4 Data 
CSD/MRA&L  (Rm.  3E919) 

The  Pentagon 
Washington,  DC  20301 

1 Mr.  Fredrick  W.  Suffa 
MPP  (A&R) 

2E269 

Pentagon 

Washington,  D.C.  20301 
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1 Dr.  Susan  Chipman 
Basic  Skills  Program 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20205 


PROF.  EARL  A.  ALLUiSl 
DEPT.  OE  PSYCHOLOGY 
CODE  237 

OLD  DOMINION  UNIVERSITY 
NORFOLK , VA  23505 


1 Dr.  William  Gorham,  Director 
Personnel  R*D  Center 
U.S.  Civil  Service  Commission 
1 9°0  E Street  NW 
Washington,  DC  20415 

1 Dr.  Vern  W.  Urry 

Personnel  R4D  Center 
U.S.  Civil  Service  Commission 
1900  E Street  NW 
Washington,  DC  20415 

1 C.S.  WIN1EWICZ 

U.S.  CIVIL  SERVICE  COMMISSION 
REGIONAL  PSYCHOLOGIST 
2?0  S.  DEARBORN  STREET 
CHICAGO,  IL  60504 

1 Dr.  Joseph  L.  Young,  Director 
Memory  4 Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  2^550 


1 


"L 


1 Dr.  John  R.  Anderson 
Dept,  of  Psychology 
Yale  University 
Now  Haven,  CT  06520 

1 1 psychological  research  unit. 

Dept.,  of  Defense  (Army  Office) 
Campb'll  Park  Offices 
Canberra  ACT  2600,  Australia 

1 MR.  SAMUEL  PALL 

EDUCATIONAL  TESTING  SERVICE 
PRINCETON,  NJ  06540 

1 Dr.  Nicholas  A.  Lond 
Dept,  of  Psychology 
Sacramento  State  Co l Lege 
600  Jay  Street 
Sacramento,  CA  95619 

1 Dr.  David  G.  Eowe-rs 

Institute  for  Social  Research 
University  cf  Michigan 
Ann  Arbor,  MI  45106 

1 Dr.  John  Seeley  Brown 

Bolt  Earanek  4 Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  C21?8 

1 Dr.  John  P.  Carroll 
Psychometric  Lab 
Univ.  of  No.  C-rolin^ 

Davie  Hall  01?A 
Chapel  Hill,  NC  27514 

1 Dr.  William  Chase 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  1521? 
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1 Dr.  K-nneth  E . Clark 

Col  leer  of  A'-ts  l Sciences 
University  of  Po^h-st^r 
River  Campus  Station 
Rochester , NY  lUf'27 

1 Dr.  Nornan  Cliff 
Dept.,  of  Psychology 
Univ.  of  So.  California 
University  Par-; 

Los  Angeles,  CA  92007 

1 Dr.  Allan  M.  Collins 

Bol*  Perm  : k h Newman,  Ins. 

GO  Moulton  Street 
Cambridge,  Mo  021  ?8 

1 Dr.  Meredith  Crawford 
5605  Montgomery  Street 
Chevy  Chase,  MD  2001c 

1 DR.  FENE  V.  DAW IS 
DEPT.  OF  PSYCHOLOGY 
UNIV.  CF  MINNESOTA 
75  E.  RIVER  RD. 

MINNEAPOLIS,  MN  55**GG 

1 Dr.  Marvin  D,  Dunnette 
NU9?  El ’ • ' • Hall 
Dep»  . o ■ y ■'ho logy 
Univ.  of  linn  rota 
Minneapolis,  MN  55u55 

1 MAJOR  I.  N.  EVONIC 

CANADIAN  FORCES  PERS.  APPLIED  RESEARCH 
1107  AVENUE  ROAD 
TORONTO , ONTARIO,  CANADA 

1 Dr.  Richard  L.  Ferguson 

Th°  American  College  Testing  Program 
P.0.  Rox  1G3 
3 ova  City,  I A *922^0 

1 Dr.  Victor  Fields 
D^pt.  of  Psychology 
Montgomery  College 
Rockville,  MD  2"SG9 

*L 


Non  Govt 


1 Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organ. 
8agg  sixteenth  Street 
Silver  Spring,  MD  20910 

1 Dr.  John  R.  Frederiks-'-n 
Rolt  Beranek  & Newman 
50  Moulton  St ref;t. 

Cambridge,  MA  02178 

1 DR.  ROBERT  GLASER 
LRDC 

UNIVERSITY  CF  PITTSBURGH 
3939  O'HARA  STREET 
PITTSBURGH,  PA  15213 

1 DR.  JAMES  G.  GREENO 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
307q  O'HARA  STREET 
PITTSBURGH,  PA  1521? 

1 Dr.  Ron  Hambleton 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  21002 

1 Dr.  Richard  S.  Hatch 

Decision  Systems  Assoc.,  Inc. 

350  Fortune  Terrace 
Rockville,  MD  2085*1 

1 Library 

HumRRO/Western  Division 
27857  Berwick  Driv^ 

Carmel , CA  93921 

1 Dr.  Earl  Hunt 

Dept,  of  Psychology 
University  of  Washington 
Seattle,  WA  98105 

1 Mr.  Gary  Irving 

Data  Sciences  Division 
Technology  Services  Corporation 
2311  Wilshirc  Plvd. 

Santa  Monica  CA  90*103 
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Hon  Govt 


Non  Govt 


DK.  LAWRENCE  F.  JOHNSON 
LAWRENCE  JOHNSON  '<  ASSOC.,  INC. 

SUITE  502 
2001  S STREET  NV.' 

WASHINGTON,  DC  20009 

Dr.  Ezra  S.  Krendel 
W.narton  School, DH/DC 
University  of  Pennsylvania 
Philadelphia,  PA  I?1 7 9 

Hr.  Marlin  Kroner 
111?  Via  Go l^t a 

Palos  Verdes  Estates,  CA  9027E 

LCOL . C.R.J.  LA  FLEUR 
PERSONNEL  APPLIED  RESEARCH 
NATIONAL  DEFENSE  HQS 
101  COLONEL  EY  DRIVE 
OTTAWA , CANADA  K1 A 0K2 

Dr.  Frederick  M.  Lord 
Educational  Testing  Service 
Princeton,  IJJ  oCSHO 

Dr.  Robert  R.  Mackie 
Hunan  Factors  Research,  Inc. 

6780  Cortona  Drive- 
Santa  Farb’ra  Research  Pk . 

Go  lota , CA  92017 

Dr.  Donald  A Norman 
C-pt.  of  Psychology  C-099 
Univ.  of  California,  San  Diego 
Lc  Jolla,  CA  9209? 

Dr.  Jesse  Orlansky 
Institute  for  Defense  Analysis 
AlDO  Army  Navy  Drive 
Arlington,  VA  222'? 

Dr.  Seymour  A.  Paper 
Massachusetts  Insti^u^e  of  Technology 
Artificial  Tpeefiij^rce  Lib 
5^5  Technology  Square 

Cambridge,  m.P  r21?9 


MR.  LUJG1  PETRliLLC 
?U;  1 f.  EDGEWCOD  STHEEI 
ARLINGTON,  VA  22207 

DR.  STEVEN  M.  PiNE 
N660  ELLIOTT  HALL 
UNIVERSITY  OF  MINNESOTA 
75  E.  RIVER  ROAD 
MINNEAPOLIS,  MN  55"55 

DR.  PETER  POLSGN 
DEPT . OF  PSYCHOLOGY 
UNIVERSITY  OF  COLORADO 
ECULDSR,  CO  SO?, 02 

MIN . RET.  M.  RAUCH 
P II  H 

BUNDESMINISTERIUM  DER  VERTE1D1GUNG 
PCSTFACH  1 61 
5?  BONN  1,  GERMANY 

Dr.  Mark  D.  Reckase 
Educational  Psychology  Dept. 
University  of  Missouri-Columbia 
12  Hill  Hall 
Columbia,  MO  65201 

Dr.  Joseph  W.  Rigne.y 
Univ.  of  So.  California 
Behavioral  Technology  Labs 
2717  South  Hope  Street 
Los  Angeles,  CA  90007 

Dr.  Andrew  M.  Rose 
American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NV 
Washington,  DC  20007 

Dr.  Leonard  L.  Rosenbaum,  Chairman 
Department  of  Psychology 
Montgomery  College 

Rockville,  MD  20*50 

Dr.  Ernst  2.  Rothkopf 
Bell  Laboratories 
Goo  Mountain  Avenue 
Murray  Hill,  NJ  0797'! 
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1 DR.  THOMAS  WALLSTEN 

PSYCHOMETRIC  LABORATORY 
DAVIE  HALL  01 ?A 
UNIVERSITY  OF  NORTH  CAROLINA 
CHAPEL  HILL,  NC  P751 ^ 

1 Dr.  Claire  E.  Weinstein 

Educational  Psychology  Df-pt. 
Univ.  of  T<~xas  at  Austin 
Austin,  TX  7371? 

1 Dr.  David  J.  Weiss 
NC60  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55^55 

1 DR.  SUSAN  E..  WHITELY 
PSYCHOLOGY  DEPARTMENT 
UNIVERSITY  OF  KANSAS 
LAWRENCE,  KANSAS  660UU 


1 DR.  ALBERT  STEVENS 

c0LT  EERAMEK  9 NEWMAN,  INC. 

50  MOULTON  STREET 
CAMBRIDGE , MA  021*8 

1 DR.  PATRICK  SUPPES 

INSTITUTE  FC.R  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STAMFORD,  CA  9^05 

1 Dr.  Kiku.mi  Tatsuoka 

Computer  Pas^d  Education  Research 
Laboratory 

25?  Engin'-ering  Research  Laboratory 
University  of  Illinois 
Urban?.,  IL  6I801 

1 DR.  PERRY  THCRNDYKE 
THE  RAND  CORPORATION 
1700  MAIN  STREET 
SANTA  MONICA,  CA  90U06 


1 PRCF . FUMIKO  SAMEJIMA 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  TENNESSEE 
KNOXVILLE,  TN  37916 

1 DR.  WALTER  SCHNEIDER 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  CF  ILLINOIS 
CHAMPAIGN,  IL  6l8?P 

1 DR.  ROFERT  J.  SEIDEL 

INSTRUCTIONAL  TECHNOLOGY  GROUP 
HUFF.  HO 

"00  ,'J . WASHINGTON  ST. 
ALEXANDRIA,  VA  2231  *» 

i Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  aipo* 

1 Dr.  Fob'rt  Sternberg 
Dept,  of  Psychology 

iYal<'  University 

Box  11A,  Yal<^  Station 
New  Haven,  CT  06520 


